Aligning Two Images By Matching Their Feature Points

ABSTRACT

A multiple camera imaging system, comprising: a first camera image sensor configured to obtain a first image of a scene from a first vantage perspective point; a second camera image sensor configured to obtain a second image of the scene from a second vantage perspective point; and an image signal processor (ISP), configured to process the first image and the second image by performing the following steps: producing a first roughly-aligned image from the first image, and a second roughly-aligned image from the second image; using the first and second roughly-aligned images to produce a disparity image; identifying a feature point within the first roughly-aligned image; utilizing the disparity image to create a search zone within the second roughly-aligned image; and identifying a group of candidate feature points within the search zone; identifying within the search zone a best-matched candidate feature point that best matches the feature point within the first roughly-aligned image to form a best-matched feature point pair; and using information from the best-matched feature point pair to further align the first and second roughly-aligned images.

TECHNICAL FIELD

This disclosure relates to image signal processing of a dual or multiplecamera imaging system, which includes two or more cameras, with eachcamera taking its own image of the same scene from its own perspectivevantage point. More particularly, this disclosure relates to aligningtwo images by matching the feature points of the image taken by onecamera to the feature points of the image taken by another camera.

BACKGROUND INFORMATION

An array camera includes an array of individual cameras, and isalternatively referred to as a multiple camera imaging system. Anexample of such an imaging system is a dual camera system that isbecoming a popular product feature in mobile phones. Typically, theindividual cameras cooperate to provide imaging functionality thatcannot be achieved by using only one camera by itself. For example, instereo imaging, two individual cameras each takes an image of the samescene from two slightly different vantage points, thereby producing adepth perception functionality that is not achievable with a singlecamera alone. As another example, in dynamic zooming, the dual camerasystem includes a telephoto lens camera with a narrower but more focusedfield of view (FOV), and a wide FOV camera with a wider but less focusedfield of view. These two cameras are directed to each take an image ofessentially the same scene, with the telephoto lens camera providing amore zoomed-in view of the scene. The pair of images captured by thesetwo cameras may be processed and then combined to provide a range ofzoom levels, thereby producing a dynamic zooming functionality. Withonly a single camera, such functionality would require a complex,active-type mechanical adjustment of a variable imaging objective.

The abovementioned dual camera system operations rely on propercombination or superposition of two images captured by two differentcameras that are placed at slightly different positions, thus havingslightly different perspective views of the same scene. Prior to imagecombination or superposition, geometrical corrections are applied to thecaptured images to rectify each image and to attempt to align them witheach other. Conventionally, the requisite alignment process is based oncomparing pixel values between individual images to find correspondingpixels. However, this alignment process is offline, meaning that it isdone at a time when a real image is not being taken by the dual camerasystem, and usually takes place before the camera product is shipped tothe customer. There are online residual errors that may causemisalignment when the dual camera system is being used to capture realimages. Such residual errors cannot be corrected by an offline alignmentprocess.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive examples of the invention are describedwith reference to the following figures, wherein like reference numeralsrefer to like parts throughout the various views unless otherwisespecified.

FIG. 1 is a schematic diagram showing a first camera sensor imaging ascene to produce a first image from a first perspective vantage point,and a second camera sensor imaging the same scene to produce a secondimage from a second perspective vantage point.

FIG. 2 is a diagram showing a first roughly-aligned image containing asingle feature point, and a second roughly aligned image containing agroup of candidate feature points throughout the image, wherein thecandidate feature points may be used to match the feature point in thefirst roughly-aligned image.

FIG. 3 is a diagram showing the utilization of the first and secondroughly-aligned images to produce a disparity image.

FIG. 4 is a diagram showing the utilization of the disparity image toform a search zone within the second image, wherein the search zonecontains a group of candidate feature points that may be used to matchthe feature point in the first roughly-aligned image.

FIG. 5 is a flow chart showing a multitude of digital image signalprocessing blocks and their operations within an image signal processor(ISP) of a multiple camera imaging system.

FIG. 6 is a schematic diagram showing a multiple camera imaging systemthat includes a first camera sensor to image a scene to produce a firstimage from a first perspective vantage point, a second camera sensor toimage the same scene to produce a second image from a second perspectivevantage point, and an image signal processor (ISP) to process and alignthe first and second images.

Corresponding reference characters indicate corresponding componentsthroughout the several views of the drawings. Skilled artisans willappreciate that elements in the figures are illustrated for simplicityand clarity and have not necessarily been drawn to scale. For example,the dimensions of some of the elements in the figures may be exaggeratedrelative to other elements to help to improve understanding of variousembodiments of the present invention. Also, common but well-understoodelements that are useful or necessary in a commercially feasibleembodiment are often not depicted in order to facilitate a lessobstructed view of these various embodiments of the present invention.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth toprovide a thorough understanding of the examples. One skilled in therelevant art will recognize; however, that the techniques describedherein can be practiced without one or more of the specific details, orwith other methods, components, materials, etc. In other instances,well-known structures, materials, or operations are not shown ordescribed in detail to avoid obscuring certain aspects.

Reference throughout this specification to “example” or “embodiment”means that a particular feature, structure, or characteristic describedin connection with the example is included in at least one example ofthe present invention. Thus, the appearances of “example” or“embodiment” in various places throughout this specification are notnecessarily all referring to the same example. Furthermore, theparticular features, structures, or characteristics may be combined inany suitable manner in one or more examples.

Throughout this specification, several terms of art are used. Theseterms are to take on their ordinary meaning in the art from which theycome, unless specifically defined herein or the context of their usewould clearly suggest otherwise.

Multiple Camera Imaging System and Disparity Direction

FIG. 6 is a schematic diagram showing a multiple camera imaging system600 that includes a first camera (or camera sensor) 601, a second camera(or camera sensor) 602, and an image signal processor (ISP) 650. Theimaging system 600 takes images of a scene 610. More specifically, thefirst camera 601 obtains a first image 611 of the scene 610 from a firstvantage point (for example, the left side). The second camera 602obtains a second image 612 of the same scene 610 from a second vantagepoint (for example, the right side). It is appreciated that the firstimage 611 and the second image 612 have different perspectives. Bothimages are conveyed to the ISP 650 to be processed, the operation ofwhich may include some or all the steps of rough alignment (cropping,scaling, etc.), monotone conversion, further alignment, and otherrelevant image signal processing techniques. After processing, theimaging system 600 outputs a first processed image 661, and a secondprocessed image 662 that is aligned (but not necessarily matchedexactly) with the first processed image 661. These two processed imagesmay then be used for various applications, such as depth perception,dynamic zooming, etc.

FIG. 1 is a diagram that shows more details of the aforementionedmultiple camera imaging system, particularly the details relating to therelative spatial relationship of the two cameras therein. As shown inFIG. 1, a multiple camera imaging system 100 includes a first camera (orcamera sensor) 101, and a second camera (or camera sensor) 102. In thepresent embodiment, the first and second cameras 101 and 102 appear tohave the same intrinsic properties. It is appreciated that in anotherembodiment, the two cameras may have different intrinsic properties. Forexample, the first camera 101 may be a telephoto lens camera, whereasthe second camera 102 may be a wide field-of-view (FOV) camera.

The first camera 101 produces a first image 111 of a scene 100 from afirst perspective vantage point (for example, the left side). The secondcamera 102 produces a second image 112 of the same scene 100 from asecond perspective vantage point (for example, the right side). It isappreciated that the first image 111 and the second image 112 havedifferent perspectives. In addition, in a exemplary scenario not shownin FIG. 1, when the first camera 101 is a telephoto lens camera, and thesecond camera 102 is a wide FOV camera, the second image 112 may containmost or all the objects in the first image 111, plus additional objectsnot in the first image 111, due to the wide FOV property of the secondcamera 102.

As shown in FIG. 1, the first and second cameras 101 and 102 arepositioned along a first direction 103 (also marked as direction D),which directly connects the first and second cameras 101 and 102. Thisfirst direction 103 is also known as a disparity direction, because itis along this direction 103 that the first and second images 111 and 112are different in perspective, even though both images are taken of thesame scene 110. A basic cause for this disparity is that there is apositional difference between the first and second cameras 101 and 102along this first direction 103. There are several aspects to thisdisparity.

As an illustrative example for the first aspect of the disparity, thescene 110 includes scene objects 110A, 110B, 110C, 110D, and 110E, asshown in FIG. 1. Due to the first vantage point of the first camera 101,it takes the first image 111 that includes first image objects 111A,111B, 111C, and 111E, which correspond to scene objects 110A, 110B,110C, and 110E. Scene object 110D is not captured in the first image111. On the other hand, due to the second vantage point of the firstcamera 102, it takes the second image 112 that includes second imageobjects 112A, 112B, 112C, and 112D, which correspond to scene objects110A, 110B, 110C, and 110D. Scene object 110E is not captured in thesecond image 112. This type of disparity between the first and secondimages 111 and 112 primarily exists only along the first direction 103.This aspect of disparity may be rectified in a preliminary imageprocessing step (disclosed later) by cropping the first and secondimages 111 and 112 to contain essentially the same objects in the firstdirection 103. As shown in FIG. 1, a first cropped (or roughly-aligned)image 113, which is cropped from the first image 111, containsessentially the same objects as a second cropped (or roughly-aligned)image 114, which is cropped from the second image 112. Cropping helps torender the first and second images 111 and 112 to be more similar toeach other than their original raw forms. It is part of a roughalignment process.

In addition, there is another aspect of the disparity. Morespecifically, the relative spatial relationships between the imageobjects within the first and second images 111 and 112 are different.For example, both the scene objects 110A and 110B are present in thefirst image 111 as first image objects 111A and 111B, and are alsopresent in the second image 112 as second image objects 112A and 112B.However, the spatial relationship between image objects 111A and 111B inthe first image 111 appear to be different (in FIG. 1, they appear to becloser, for example) than the spatial relationship between image objects112A and 112B in the second image 112. An exemplary reason of thisaspect of disparity is parallax, which cannot be removed by cropping orother similar techniques.

In the example above, it is presumed that the first and second cameras101 and 102 face directly forward toward the scene 110. In a differentscenario not shown in FIG. 1, if the first camera 101 on the left sideslants to the right side, and if the second camera 102 on the right sideslants to the left side, then the scene objects captured by each camerawill be different from the example in FIG. 1. A person of ordinary skillin the art will be able to appreciate this different scenario, and itsramifications with regard to the various aspects of disparity asdisclosed above, hence further details of this scenario is not disclosedherein.

A second direction 104 (also marked as direction R) is orthogonal to thefirst direction 103. Along this orthogonal direction 104, there is nopositional difference between the first and second cameras 101 and 102.Accordingly, between the first and second images 111 and 112, there isno disparity in the second direction 104. Therefore, the second,orthogonal direction 104 is not a disparity direction.

Preliminary Image Processing

After the first and second images 111 and 112 have been obtained by thefirst and second cameras 101 and 102, the two images may go through apreliminary processing step, which is based on a pre-shipping (oroff-line) calibration process. More specifically, since the positionsand the functionalities of the first and second cameras 101 and 102 areknown, the pre-shipping calibration process may be engaged to utilize acalibration chart to obtain intrinsic and extrinsic matrices anddistortion coefficients of the two cameras. This will help to rectify(e.g., cropping, as disclosed previously) the first and second images111 and 112 in aspects such as having the same field of view, and alsoto roughly align them. However, even under the best circumstances, therewill be post-shipping (or on-line) residual errors that occur when thefirst and second cameras 101 and 102 are being used to capture images inreal usage. These residual errors may be due to exemplary causes such asopen loop voice coil motor (VCM) inaccuracy, relative movement betweenthe two cameras due to vibration, alterations to the camera performancedue to usage, etc. The preliminary image processing based on thepre-shipping (or off-line) calibration process cannot correct for thesepost-shipping (or on-line) residual errors. Therefore, further imagealignment after the preliminary image processing step is needed.

The preliminary image processing step includes several sub-steps. Thegoal is to achieve rough alignment of the first and second images, andprepare them for subsequent steps to further refine their alignment.

An exemplary preliminary processing step is disclosed herein. First, oneor both of the images are cropped and/or zoomed based on pre-shipping(off-line) camera calibration data, so that they contain essentially thesame objects. As an example, in FIG. 1, the first image 111 is croppedto produce the first cropped (or roughly-aligned) image 113 thatincludes the first image objects 111A, 111B, and 111C, which correspondto the scene objects 110A, 110B, and 110C, respectively. The secondimage 112 is cropped to produce the second cropped (or roughly-aligned)image 114 that includes the second image objects 112A, 112B, and 112C,which correspond to the scene objects 110A, 110B, and 110C,respectively. The first and second cropped (or roughly-aligned) images113 and 114 therefore contain essentially the same objects. In the case(not shown in FIG. 1) where the first camera 101 is a telephoto lenscamera (with a narrower viewing range), and the second camera 102 is awide FOV camera (with a wider viewing range), the second image 112 (awide FOV image containing more objects) is cropped so that it includessubstantially the same objects as the first image 111 (a close-uptelephoto image containing fewer objects). In a preferred embodiment,the first and second cropped (or roughly-aligned) images 113 and 114 arerendered to have essentially the same objects in the disparity direction103. In contrast, there is more leeway in terms of image cropping in theorthogonal direction 104.

A zooming (or some other type of up or down sampling operation)operation may also be applicable in conjunction with cropping, in orderto render the two roughly-aligned images 113 and 114 to have essentiallythe same scale, for the ease of subsequent operations. The zoomingoperation is appropriate in an exemplary situation where one camera is atelephoto lens camera, and the other camera is a wide FOV camera.

In the description above, it is presumed that the first and secondcameras 101 and 102 face directly forward toward the scene 110. In adifferent scenario, if the first camera 101 on the left side slants tothe right side, and if the second camera 102 on the right side slants tothe left side, then the scene objects captured by each camera will bedifferent from the example above. A person of ordinary skill in the artwill be able to appreciate this different scenario, and itsramifications with regard to cropping and zooming. It is appreciatedthat the cropping-zooming operation of this scenario follows the samegoal that both cropped-zoomed images will contain substantially the sameobjects, particularly in the disparity direction, as well as the sameimage scale.

In a second sub-step of preliminary image processing, the first andsecond roughly-aligned images 111 and 112, in case they are colorimages, are converted into monotone images. A color image may haveseveral color channels, for example, red, green and blue channels.Converting a color image into a monotone image may be done in severalways. A first type of monotone image may be produced by taking the valueof only one color channel, for example, the green channel (in partbecause human eye is most sensitive to green color). A second type ofmonotone image may also be produced by weighted averaging or summing thevalues of two or more color channels, for example, by weightedlyaveraging or summing the red, green, and blue channels (i.e., the red,green, blue channel may each have its own predetermined weight when theyare being averaged or summed). This second type of monotone image isalso known as a gray scale image, because it is essentially ablack-and-white image with different shades of gray.

As a result of performing the preliminary processing step, the first andsecond images 111 and 112 are rendered as a first rough-aligned image113 and a second roughly-aligned image 114. Both the first and secondroughly-aligned images 113 and 114 have essentially the same objects dueto cropping, particularly in the disparity direction 103. They may alsohave the same scale due to zooming (if needed). Preliminary processinghelps to ensure that the first and second roughly-aligned images 113 and114 have a format that is more suitable for the subsequent steps tofurther align them.

Image Alignment Using Feature Matching

Generally speaking, aligning two images may involve matching specificfeatures (or feature points) between the two images. First, within afirst image, an image feature (e.g., a feature descriptor, a featurepoint or pixel, etc.) is identified. This feature may be in thecontextual format of an edge, a corner, a blob (region of interestpoints), a ridge, etc. Second, within a second image, a group ofcandidate features are identified. Each candidate feature must be of thesame format (edge, corner, blob, ridge, etc.) and value as the firstimage feature. Significantly, this group of second image candidatefeatures may be distributed throughout the second image. Lastly, a bruteforce matching (e.g., using k nearest neighbor, i.e., kNN algorithm) isperformed to match the group of second image candidate features to thefirst image feature. Distances between the first image feature and eachof the second image candidate features are computed and sorted. Abest-matched pair is readily identified. This process is repeatedseveral times to generate a multitude of best-matched pairs between thefirst and second images. Information from these best-matched pairs isthen used to further align the two images.

FIG. 2 is a diagram that illustrates the aforementioned traditionalbrute force approach. A first image 200 originating from the firstcamera 101 (see FIG. 1) and a second image 250 originating from thesecond camera 102 (also see FIG. 1) are presented herein. Each image maybe the result of the aforementioned preliminary image processing. Hence,both the first image 200 and the second image 250 may beroughly-aligned, monotone images, and are respectively exemplified bythe first and second roughly-aligned images 113 and 114 in FIG. 1. Thefirst roughly-aligned image 200 contains a first image object 201. Afirst image feature point or pixel, labeled as “a”, is identified andshown within the first image 200. The second roughly-aligned image 250contains a second image object 251. Both the first and second imageobjects 201 and 251 are renditions of the same scene object. Within thesecond image 250, a group of candidate feature points or pixels arelabeled “A”, “B”, through “M”, and are distributed throughout the secondimage 250 (a frequent occurrence). According to traditional brute forcematching, each candidate feature point or pixel (A, B, through M) mustbe individually assessed, until a best-matched candidate is found. Forexample, first image feature point or pixel “a” and second imagecandidate feature point or pixel “B” may be deemed to be a best-matchedpair.

This conventional brute force approach to align images is generally timeconsuming and computationally expensive, because distances between eachfeature point in the first image and each candidate feature point in thesecond image must be exhaustively calculated and sorted. For example,suppose there are 1000 feature points in the first image, and for eachfirst image feature point, there are 2000 candidate feature points inthe second image, then there needs to be two million (1000 times 2000)distance computations, plus associated sorting computations, in order toachieve the final matching result.

In a dual camera system, a first camera and a second camera are bothpointing at the same scene. The distance separating the two cameras isgenerally small, for example, several centimeters apart. Each cameratakes its image from its own unique vantage point, but vantage pointdifference between the two cameras is not very big. Therefore, a firstcamera image and a second camera image, although they are different, arenot significantly different. For any feature point in the first cameraimage, there exists a best-matched candidate feature point in the secondcamera image, but this best-matched candidate feature point should existwithin the vicinity of (i.e., not too far away from) the location of thefirst camera image feature point. Brute force matching of all candidatefeature points throughout the entire second image is generallyunnecessary and wasteful for this type of dual camera system. Thecurrent disclosure presents an approach that utilizes the fact that thetwo images produced by a dual camera system do not differ by too much.Compared with the conventional brute force approach, the currentdisclosure consumes much less time, and incurs much less computationalcost.

Disparity Image Generation

A crucial aspect of the current disclosure is the generation of adisparity image from the first and second roughly-aligned images. FIG. 3shows a first roughly-aligned image 200 and a second roughly-alignedimage 250, both of which are being used to generate a disparity image300. Various methods may be used to generate this disparity image 300,as one of ordinary skill in the art would appreciate. For example, asemi-global block matching method may be used. Generally speaking, thesetypes of stereo matching methods compute the similarity between pixelsby comparing windows around the pixels of interest. The disparity image300 is alternatively known in the art as a depth map, for the followingreason. Within the disparity image (or depth map) 300, each point orpixel has a disparity value, which is inversely related to the depth ofthat point in relation to the dual cameras. Parallax is more prominentwhen an imaged object is closer to the image sensors (e.g., dualcameras, eyes). More specifically, if a point of a scene object is closeto the dual cameras, the disparity value due to the positionaldifference of the two cameras is more prominent (due to more parallax).If a point is far away from the dual cameras, the disparity value isless prominent (due to less parallax). Accordingly, the disparity valueof each point in image 300 contains depth information.

The dotted outline in the disparity image 300 represents the generalcontour and shading of a disparity image (depth map). It is not an exactrendition of the image objects 201 and 251. Rather, as we use differentshades of gray to represent the disparity value at each point of theimage 300, with higher value represented by a darker gray shade (or alighter shade, depending on choice of convention), the overall contourand shading of image 300 will manifest a similar shape outline as imageobjects 201 and 251 of the roughly-aligned imaged 200 and 250.

Several issues are worth further elaboration. First, according to FIG.1, the first camera 101 and the second camera 102 are positioned alongthe first direction 103, which in this case is the horizontal direction(or x direction). This is also the disparity direction. Accordingly, thedisparity value at each point of the disparity image 300 is denoted asΔx, which indicates an image disparity in the x direction as caused bythe positional difference between the two cameras 101 and 102. Second,there is no disparity value in the second direction 104 (see FIG. 1),because there is no positional difference between the first and secondcameras 101 and 102 in this second direction (y direction) that isorthogonal to the x direction. Third, as an optional step, the disparityimage (or depth map) 300 may be down-sampled in order to speed up thesubsequent calculation when this image is being used to create a searchzone. For example, a 2:1 down-sampling in both the rows and the columnsof the image 300 may result in a quarter-resolution, down-sampleddisparity image 350, as shown in FIG. 3.

Identifying a Feature Point and a Group of Candidate Feature Points

Another crucial aspect of the current disclosure is selecting featurepoints in the first and second roughly-aligned images 200 and 250, andidentifying best-matched pairs between these two sets of feature points.Further details are disclosed in the following.

First, as shown in FIG. 4, a first image feature “a” is identified inthe first roughly-aligned image 200. This may be a feature point, pixel,or some other type of descriptor. Several such features may beidentified in the entire process. For the sake of simplicity inillustration, only one such first image feature “a” is discussed, and ishereinto regarded as one feature point. A number of selection criteriamay be applied to select the first image feature point “a”. For example,an edge, a corner, a ridge, etc., may be considered in selecting thefirst image feature point “a”. For the sake of illustration, theposition of this feature point “a” in image 200 may be represented by anordered pair Cartesian system notation (x_(a), y_(a)).

Second, also shown in FIG. 4, a disparity image (depth map) 300 isconsidered in conjunction with the first image feature point “a” inorder to identify an associated disparity value. A down-sampleddisparity image 350 (see FIG. 3) may be similarly used (e.g., aftersame-resolution up-sampling) as the disparity image 300. As previouslyexplained, the disparity image 300 contains disparity information thatis denoted as Δx, which indicates an image disparity only in the xdirection as caused by the positional difference between the two cameras101 and 102. In the current example, within the disparity image 300, apoint position represented by “X” has the same location (x_(a), y_(a))as point “a” within the first roughly-aligned image 200. At point “X”within the disparity image 300, the specific disparity value associatedwith (x_(a), y_(a)) may be looked up. This is the associated disparityvalue that will be useful in the next step. This disparity value may bedenoted as Δx_(a), and is alternatively referred to as a depth vector.It is appreciated that there is no disparity value in the y direction,because there is no positional difference between cameras 101 and 102 inthe y direction 104 (see FIG. 1).

Third, the first image feature point position (x_(a), y_(a)) and itsassociated disparity value Δx_(a) may be combined to create a newposition (x_(a)+Δx_(a), y_(a)) that will serve as the center of a searchzone (or search window) 400 within the second roughly-aligned image 250,as shown in FIG. 4. This search zone (or search window) 400 is sometimesreferred to as a geometric prior. Its shape may be rectangular, or someother shapes (such as some regular polygonal shape, or a circle). Whenthe search zone is a rectangle, it may have a size of m by n, wherein mand n represent number of pixels. For example, both m and n may be 41pixels.

Fourth, as shown in FIG. 4, within the search zone 400 of the secondroughly-aligned image 250, a group of candidate feature points “A”, “B”,through “E” may be identified. Each of these candidate feature pointswill then be individually assessed to match the first image featurepoint “a”. Since the search zone 400 is much smaller than the entireimage 250, e.g., 41×41 pixels (search zone) vis-à-vis 1080×1440 pixels(entire image), the number of candidate feature points within the searchzone is significantly less. This may be exemplarily appreciated bycomparing FIGS. 2 and 4. In FIG. 2, there are many candidate featurepoints (A, B, through M) within the entire roughly-aligned image 250. Incontrast, in FIG. 4, there are fewer candidate feature points (A, B,through E) within the search zone 400, which is itself contained withinthe second roughly-aligned image 250. The reduction of candidate featurepoints (usually on the order of 100 times or more) is a key reason forthe time and computational cost saving of the image alignment system ofthe current disclosure.

Identifying the Best Candidate Feature Point to Match the Feature Point

Subsequently, a local brute force matching may be performed to compareeach candidate feature point (A, B, through E) within the search zone400 with the first image feature point “a”. For example, a kNN algorithmmay be employed. Distances between the feature point “a” and eachcandidate feature point (A, B, through E) may be computed and sorted. Abest-matched pair may be identified accordingly. For the sake ofillustration, suppose candidate feature point “B” is best matched to thefeature point “a”. Then the best-matched pair information would includethe Cartesian notation (x_(a), y_(a)) and (x_(B), y_(B)) of these twopoints.

The aforementioned process may be repeated several times to identifymore best-matched pairs. For each time of the repetition, a featurepoint is identified within the first image 200; an associated,corresponding disparity value is identified with the help of thedisparity image 300; a search zone 400 is created accordingly within thesecond image 250; a group of candidate feature points are identifiedwithin this search zone 400; and a local brute force matching isemployed to identify a best-matched pair.

Modeling and Rectification Operations

After local matching, a list of best-matched pairs will be identified.Information contained in these best-matched pairs may then be used tofurther align the first and second roughly-aligned images 200 and 250. Anumber of rectification models may be employed, as appreciated by one ofordinary skill in the art. As an example, an affine model with scalingparameter and shift parameter may be used in the modeling. In addition,based on an online calibration model, linear interpolation may be usedto rectify one or both of the images. The end result is to produce thefirst and second processed images 661 and 662, as previously shown inFIG. 6. These images are now better aligned with each other, and aresuitable for further information extraction and/or image processing,such as depth perception, dynamic zooming, etc.

Exemplary Image Signal Processing Operation Flow

An exemplary image signal processing operation flow is disclosed hereinto restate and emphasize some aspects of the image processingembodiments as described above. This is shown in FIG. 5, wherein eachrectangular block stands for a processing block, and each roundedrectangle stands for a data form, such as an image, a value, a searchzone, a feature point, a pair of matched feature points, etc.

As shown in FIG. 5, an image processing method 500 starts with using afirst and second image sensor cameras to obtain, from differentperspective vantage points, a first image 501 and a second image 502,and sending both images to a preliminary processing block 510. Thepreliminary processing block 510 performs several tasks: (1) selectivecropping based on pre-defined, intrinsic and extrinsic camera properties(position, field of view, magnification, etc.), as performed by amandatory rough alignment block 515; (2) optional zooming operation asneeded, as performed by an optional image scaling block 516; and (3)optional conversion from color to monotone as needed, as performed by anoptional monotone block 517. As a result, the preliminary processingblock 510 produces a first roughly-aligned image 511 and a secondroughly-aligned image 512. It is noteworthy that the first and secondcameras are positioned along a disparity direction, such as the xdirection. In FIG. 1, this disparity direction is exemplarilyrepresented as the horizontal direction 113. There is no disparity inthe y direction 114, which is orthogonal to the horizontal direction113, as shown in FIG. 1.

These two roughly-aligned images are then sent to a disparity imagecreation block 520, which produces a disparity image (or depth map) 521.Methods such as semi-global block matching may be used to create thedisparity image 521. It is noteworthy that the disparity image 521contains disparity information only in the disparity direction, such asthe x direction. In the orthogonal y direction, there is no disparityinformation.

A first feature (or feature point) selector block 530 may be used toselect a first image feature point 531 within the roughly-aligned image511. This first image feature point 531 may have an exemplary positionat (x_(a), y_(a)). Also, the same position is found within the disparityimage 521, and an associated disparity value (depth vector) 541 islooked up. For example, the disparity value 541 may be represented asΔx_(a). This step is performed by a disparity value generator block 540.Next, the position of the first image feature point 531 (x_(a), y_(a))may be combined with the disparity value Δx_(a) to generate the centerposition (x_(a)+Δx_(a), y_(a)) of a search zone (or search window) 551.This step is performed by a search zone generator block 550. The searchzone 551 is also known as a geometric prior, and covers an area that ismuch smaller than the second roughly-aligned image 512. For example, thesearch zone 551 may be 41×41 pixels, whereas the entire secondroughly-aligned image 512 may be 1080×1044 pixels.

A second feature (or feature point) selector block 560 selects a groupof candidate feature (or feature points) 561 from within the secondroughly-aligned image 512. Crucially, this group of candidate featurepoints 561 is selected only from the search zone 551, which is itselfpositioned inside the second roughly-aligned image 512. The center ofthe search zone 551 is at (x_(a)+Δx_(a), y_(a)).

A feature point matching block 570 takes the first image feature point531 and the group of second image candidate feature points 561, and useslocal matching (such as kNN brute force matching) to identify abest-matched candidate feature point. These two feature points form abest-matched feature point pair 571.

The process performed by blocks 530, 540, 550, 560, and 570 may berepeated several times in order to identify a multitude of best-matchedfeature point pairs.

The information contained within these best-matched feature point pairsis then aggregated and sent to a modeling-rectification block 580, whichmay itself include model estimator operation andrectification/interpolation sub-blocks (not shown). As an example, themodeling-rectification block 580 may use interpolation to refine one orboth of the two roughly-aligned images. For example, the secondroughly-aligned image 512 is further aligned with the firstroughly-aligned image 511. The end result is a first processed image581, and a second processed image 582 that is aligned with the firstprocessed image 581. The two better aligned images 581 and 582 may thenbe used for further information extraction and/or image processing, suchas depth perception, dynamic zooming, etc.

Compared with the conventional image processing that uses brute force tomatch all the key feature points within both images, the aforementionedapproach of utilizing a disparity image (depth map) is much less costly,in terms of hardware complexity and processing speed. Experiments haveshown that whereas the conventional method requires more than 68milliseconds to obtain the appropriate affine model (before the finaltwo-dimensional image alignment), the currently disclosed method onlyrequires about 23 milliseconds. Hardware complexity in terms of buffersize is also reduced.

The above description of illustrated examples of the invention,including what is described in the Abstract, is not intended to beexhaustive or to limit the invention to the precise forms disclosed.While specific examples of the invention are described herein forillustrative purposes, various modifications are possible within thescope of the invention, as those skilled in the relevant art willrecognize.

These modifications can be made to the invention in light of the abovedetailed description. The terms used in the following claims should notbe construed to limit the invention to the specific examples disclosedin the specification. Rather, the scope of the invention is to bedetermined entirely by the following claims, which are to be construedin accordance with established doctrines of claim interpretation.

What is claimed is:
 1. A multiple camera imaging system, comprising: afirst camera image sensor configured to obtain a first image of a scenefrom a first vantage perspective point; a second camera image sensorconfigured to obtain a second image of the scene from a second vantageperspective point; and an image signal processor (ISP), configured toprocess the first image and the second image by performing the followingsteps: (a) producing a first roughly-aligned image from the first image,and a second roughly-aligned image from the second image; (b) using thefirst and second roughly-aligned images to produce a disparity image;(c) identifying a feature point within the first roughly-aligned image;utilizing the disparity image to create a search zone within the secondroughly-aligned image; and identifying a group of candidate featurepoints within the search zone; (d) identifying within the search zone abest-matched candidate feature point that best matches the feature pointwithin the first roughly-aligned image to form a best-matched featurepoint pair; and (e) using information from the best-matched featurepoint pair to further align the first and second roughly-aligned images.2. The multiple camera imaging system of claim 1, wherein one of thefirst and second camera sensors is configured to have a wider viewingrange than the other camera sensor.
 3. The multiple camera imagingsystem of claim 1, wherein the first and second images are color imagesthat include a multitude of color channels, and wherein step (a) furtherincludes the ISP averaging the multitude of color channels of the firstand second images to produce the first and second roughly-alignedimages.
 4. The multiple camera imaging system of claim 3, wherein step(a) further includes the ISP using cropping to produce the first andsecond roughly-aligned images, wherein the second roughly-aligned imagehas substantially the same objects as the first roughly-aligned image.5. The multiple camera imaging system of claim 1, wherein the firstcamera sensor and the second camera sensor are positioned along a firstdirection; wherein the first direction is perpendicular to a seconddirection; and wherein the disparity image includes disparityinformation only in the first direction.
 6. The multiple camera imagingsystem of claim 5, wherein step (b) further includes the ISPdown-sampling the disparity image to produce a lower-resolutiondisparity image, and wherein step (c) further includes the ISP utilizingthe lower-resolution disparity image to create the search zone withinthe second roughly-aligned image.
 7. The multiple camera imaging systemof claim 1, wherein the ISP repeats steps (c) and (d) to form amultitude of best-matched feature point pairs, and wherein step (e)includes using information from the multitude of best-matched featurepoint pairs to further align the first and second roughly-alignedimages.
 8. A method of digital image signal processing, comprising: (a)providing a multiple camera imaging system including a first camerasensor, a second camera sensor, and an image signal processor (ISP); (b)using the first camera sensor to obtain a first image of a scene from afirst vantage perspective point; using the second camera sensor toobtain a second image of the scene from a second vantage perspectivepoint; (c) using the ISP to produce a first roughly-aligned image fromthe first image, and a second roughly-aligned image from the secondimage; (d) using the ISP to utilize the first and second roughly-alignedimages to produce a disparity image; (e) using the ISP to identify afeature point within the first roughly-aligned image; to utilize thedisparity image to create a search zone within the secondroughly-aligned image; and to identify a group of candidate featurepoints within the search zone; (f) using the ISP to identify within thesearch zone a best-matched candidate feature point that best matches thefeature point within the first roughly-aligned image to form abest-matched feature point pair; and (g) using the ISP to utilizeinformation from the best-matched feature point pair to further alignthe first and second roughly-aligned images.
 9. The digital image signalprocessing method of claim 8, wherein one of the first and second camerasensors is configured to have a wider viewing range than the othercamera sensor.
 10. The digital image signal processing method of claim8, wherein the first and second images are color images that include amultitude of color channels, and wherein step (c) further includes theISP averaging the multitude of color channels of the first and secondimages to produce the first and second roughly-aligned images.
 11. Thedigital image signal processing method of claim 10, wherein step (c)further includes the ISP using cropping to produce the first and secondroughly-aligned images, wherein the second roughly-aligned image hassubstantially the same objects as the first roughly-aligned image. 12.The digital image signal processing method of claim 8, wherein the firstcamera sensor and the second camera sensor are positioned along a firstdirection; wherein the first direction is perpendicular to a seconddirection; and wherein the disparity image includes disparityinformation only in the first direction.
 13. The digital image signalprocessing method of claim 13, wherein step (d) further includes the ISPdown-sampling the disparity image to produce a lower-resolutiondisparity image, and wherein step (e) further includes the ISP utilizingthe lower-resolution disparity image to create the search zone withinthe second roughly-aligned image.
 14. The digital image signalprocessing method of claim 8, wherein the ISP repeats steps (e) and (f)to form a multitude of best-matched feature point pairs, and whereinstep (g) includes the ISP using information from the multitude of thebest-matched feature point pairs to further align the first and secondroughly-aligned images.
 15. A multiple camera imaging system,comprising: a first camera image sensor configured to obtain a firstimage of a scene from a first vantage perspective point; a second cameraimage sensor configured to obtain a second image of the scene from asecond vantage perspective point; and an image signal processor (ISP),including: (a) a preliminary image processing block configured toproduce a first roughly-aligned image from the first image, and a secondroughly-aligned image from the second image; (b) a disparity imagecreation block configured to produce a disparity image by using thefirst and second roughly-aligned images; (c) a first feature pointselector block configured to identify a feature point within the firstroughly-aligned image; (d) a disparity value generator block configuredto produce a disparity value by using the feature point and thedisparity image; (e) a search zone generator block configured toproduce, within the second roughly-aligned image, a search zone by usingthe feature point and the disparity value; (f) a second feature pointselector block configured to identify a group of candidate featurepoints within the search zone; (g) a feature point matching blockconfigured to identify, within the search zone, a best-matched candidatefeature point that best matches the feature point within the firstroughly-aligned image to form a best-matched feature point pair; whereinthe ISP is further configured to use information from the best-matchedfeature point pair to further align the first and second roughly-alignedimages.
 16. The multiple camera imaging system of claim 15, wherein oneof the first and second camera sensors is configured to have a widerviewing range than the other camera sensor.
 17. The multiple cameraimaging system of claim 15, wherein the first and second images arecolor images that include a multitude of color channels, and wherein thepreliminary processing block is further configured to average themultitude of color channels of the first and second images to producethe first and second roughly-aligned images.
 18. The multiple cameraimaging system of claim 17, wherein the preliminary processing block isfurther configured to use cropping to produce the first and secondroughly-aligned images, wherein the second roughly-aligned image hassubstantially the same objects as the first roughly-aligned image. 19.The multiple camera imaging system of claim 15, wherein the first camerasensor and the second camera sensor are positioned along a firstdirection; wherein the first direction is perpendicular to a seconddirection; and wherein the disparity image includes disparityinformation only in the first direction.
 20. The multiple camera imagingsystem of claim 19, wherein the disparity image creation block isfurther configured to down-sample the disparity image to produce alower-resolution disparity image; and wherein the disparity valuegenerator block is configured to produce the disparity value by usingthe feature point and the lower-resolution disparity image.
 21. Themultiple camera imaging system of claim 15, wherein the ISP repeatedlyuses the first feature point selector block, the disparity valuegenerator block, the search zone generator block, the second featurepoint selector block, and the feature point matching block to form amultitude of best-matched feature point pairs; and uses information fromthe multitude of the best-matched feature point pairs to further alignthe first and second roughly-aligned images.